Achieving seamless viewing of high-fidelity volumetric video, comparable to 2D video experiences, remains an open challenge. Existing volumetric video compression methods either lack the flexibility to adjust quality and bitrate within a single model for efficient streaming across diverse networks and devices, or struggle with real-time decoding and rendering on lightweight mobile platforms. To address these challenges, we introduce 4DGCPro, a novel hierarchical 4DGaussian compression framework that facilitates real-time mobile decoding and high-quality rendering via progressive volumetric video streaming in a single bitstream. Specifically, we propose a perceptually-weighted and compression-friendly hierarchical 4D Gaussian representation with motion-aware adaptive grouping to reduce temporal redundancy, preserve coherence, and enable scalable multi-level detail streaming. Furthermore, we present an end-to-end entropy-optimized training scheme, which incorporates layer-wise rate-distortion (RD) supervision and attribute-specific entropy modeling for efficient bitstream generation. Extensive experiments show that 4DGCPro enables flexible quality and multiple bitrate within a single model, achieving real-time decoding and rendering on mobile devices while outperforming existing methods in RD performance across multiple datasets. The corresponding author is Qiang Hu(qiang.hu@sjtu.edu.cn)

This paper addresses the challenge of representing high-fidelity volumetric videos with low storage cost. Some recent feature grid-based methods have shown superior performance of fast learning implicit neural representations from input 2D images. However, such explicit representations easily lead to large model sizes when modeling dynamic scenes. To solve this problem, our key idea is reducing the spatial and temporal redundancy of feature grids, which intrinsically exist due to the self-similarity of scenes. To this end, we propose a novel neural representation, named dynamic codebook, which first merges similar features for the model compression and then compensates for the potential decline in rendering quality by a set of dynamic codes. Experiments on the NHR and DyNeRF datasets demonstrate that the proposed approach achieves state-of-the-art rendering quality, while being able to achieve more storage efficiency.

Large-scale neuronal activity recordings with fluorescent calcium indicators are increasingly common, yielding high-resolution 2D or 3D videos. Traditional analysis pipelines reduce this data to 1D traces by segmenting regions of interest, leading to inevitable information loss. Inspired by the success of deep learning on minimally processed data in other domains, we investigate the potential of forecasting neuronal activity directly from volumetric videos. To capture long-range dependencies in high-resolution volumetric whole-brain recordings, we design a model with large receptive fields, which allow it to integrate information from distant regions within the brain. We explore the effects of pre-training and perform extensive model selection, analyzing spatio-temporal trade-offs for generating accurate forecasts. Our model outperforms trace-based forecasting approaches on ZAPBench, a recently proposed benchmark on whole-brain activity prediction in zebrafish, demonstrating the advantages of preserving the spatial structure of neuronal activity.

Volumetric video based on Neural Radiance Field (NeRF) holds vast potential for various 3D applications, but its substantial data volume poses significant challenges for compression and transmission. Current NeRF compression lacks the flexibility to adjust video quality and bitrate within a single model for various network and device capacities. To address these issues, we propose HPC, a novel hierarchical progressive volumetric video coding framework achieving variable bitrate using a single model. Specifically, HPC introduces a hierarchical representation with a multi-resolution residual radiance field to reduce temporal redundancy in long-duration sequences while simultaneously generating various levels of detail. Then, we propose an end-to-end progressive learning approach with a multi-rate-distortion loss function to jointly optimize both hierarchical representation and compression. Our HPC trained only once can realize multiple compression levels, while the current methods need to train multiple fixed-bitrate models for different rate-distortion (RD) tradeoffs. Extensive experiments demonstrate that HPC achieves flexible quality levels with variable bitrate by a single model and exhibits competitive RD performance, even outperforming fixed-bitrate models across various datasets.

Learn more about what comes next. Today 8i showed off its technology for quickly rendering volumetric video for real-time hologram images. The Venice, California-based company makes tools that capture, transform, and stream holograms using volumetric or 3D video. Historically, rendering volumetric video could take days, even weeks, to process a worthy asset. However, with 8i's advancement in machine learning and computer vision, this once arduous process can now be completed within milliseconds, enabling people to be broadcast live in 3D.

This paper introduces Deep4D a compact generative representation of shape and appearancefrom captured 4D volumetric video sequences of people. 4D volumetric video achieves highlyrealistic reproduction, replay and free-viewpoint rendering of actor performance from multipleview video acquisition systems. A deep generative network is trained on 4D video sequencesof an actor performing multiple motions to learn a generative model of the dynamic shapeand appearance. We demonstrate the proposed generative model can provide a compactencoded representation capable of high-quality synthesis of 4D volumetric video with two ordersof magnitude compression. A variational encoder-decoder network is employed to learn anencoded latent space that maps from 3D skeletal pose to 4D shape and appearance. Thisenables high-quality 4D volumetric video synthesis to be driven by skeletal motion, includingskeletal motion capture data. This encoded latent space supports the representation of multiplesequences with dynamic interpolation to transition between motions. Therefore we introduceDeep4D motion graphs, a direct application of the proposed generative representation. Deep4Dmotion graphs allow real-tiome interactive character animation whilst preserving the plausiblerealism of movement and appearance from the captured volumetric video. Deep4D motion graphsimplicitly combine multiple captured motions from a unified representation for character animationfrom volumetric video, allowing novel charact...

When Los Angeles-based actress and interior designer Ashley Martin Scott responded to a casting call for "mom and baby" back in April 2015, details were scant. "I pretty much came into it blindly, not knowing what to expect, aside from'a mom and a baby,'" Scott said. "And that it was something about leaving a message for your child in the future and I thought, 'That sounds fun.'" There is tech for tech's sake, and then there's tech that alters or enhances the human experience. In the second season of the Verge video series Next Level, senior editor Lauren Goode takes you behind the scenes to show you the technology that's being worked on at some of the world's most innovative companies and research institutions.